COMMENT ⊗   VALID 00005 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	.NEWSEC EXTENSIONS TO AL,VISUAL FEEDBACK
C00013 00003	.newss DYNAMIC FRAMES 
C00019 00004	.NEWSS EXTENSIONS TO OTHER ARMS AND DEVICES
C00021 00005	.NEWSS FINE CONTROL
C00023 ENDMK
C⊗;
.NEWSEC EXTENSIONS TO AL,VISUAL FEEDBACK

.NEWSS INCORPORATING VISUAL FEEDBACK
.NEWSSS NECESSARY CAPABILITIES

	This is a list of capabilities which would have to be implemented
in order to do dynamic visual feedback within AL.

  PICTURE BUFFERS AND WINDOWS

	We need a new datatype, PICTURE, to contain a digitized picture,
information on the camera used (particularly its location and orientation),
what lens was used, what filters, and perhaps other information.

Subpictures, that is windows, should be extractible from the picture itself,
so that a visual processing routine can look at whatever part it needs.

 CAMERA CONTROL

	There should be a syntax for specifying how to move a camera
    to a desired location.  For example,
.UNFILL
	AIM CAMERA_1 AT VECTOR(30,40,10)
		USING LENS=2, FILTER=CLEAR, IRIS=2.8;
.REFILL

	There should be a syntax for specifying that a picture be taken
    and stored in a certain picture buffer.  Since cameras have their own
    built-in synchronization, detailed timing control may be complicated.
    Read the %4explicit control of timing%* section below for some more ideas
    on this subject.

 OBJECT MODELS

	There should be sufficiently powerful data structures (such as
    arrays and list structures) available to implement complex object
    descriptions such as a network of features.

	It should be possible to implement programs which find
predicted objects in pictures by use of modeling information.
This may involve the use of recursion and backup, neither of
which is currently available.

  VISUAL PROCESSING PRIMITIVES

	There should be a mechanism for calling PDP11 and SPS41
    routines which share data such as pictures and object models.
(The SPS41 is a signal processor which we will use for some vision
work.)
To an extent, this already exists with the EXTERNAL MINI procedure.

MOTIONS OF ACCOMMODATION

	There should be a way of specifying how to servo an arm
    based upon visual, force, or tactile information.
The arm is expected to change its trajectory as a function of
sensory input; this would allow visual servoing, for example.
An implementation would
    involve dynamically changing the arm's destination or dynamically
    specifying relative changes to be made.  In either case, time is
    an important variable.  Consider a typical sequence of events:
.BEGIN PREFACE 0; NARROW 8,8;TABBREAK
	    (1) A picture is taken of the arm.

	    (2) The picture is analyzed to determine an arm correction.

	    (3) While the visual processing is being done, the arm
		continues to move.  Hence a prediction should be made
		and incorporated into the specified correction.

	    (4) The correction is sent to the servo.
.END

 EXPLICIT CONTROL OF TIMING

	As pointed out above, time is an important factor within
visual feedback.  It will be necessary to have "picture ready" events
which occur when data are ready for processing; it might be desirable
to allow explicit timing and scheduling to make efficient use of the
camera. 

	It would also be useful to separate the `setup' for a MOVE
from the actual beginning of a move.  This suggests a setup and
trigger mechanism to squeeze as much processing as possible into
"free" PDP11 time. 

 INTERACTIVE DESIGN OF VISUAL PROCESSING

	There should be an interface to a graphics system such as
    Bruce Baumgart's GEOMED [Baumgart] so the user can symbolically position
    the assembly parts and cameras, simulate arm motions, and 
    extract potential object models from synthetic pictures.
    The system supervisor should be flexible enough to allow the
    user to interactively manipulate the actual arms and cameras
    so that the resulting TV pictures correspond with the synthetic
    views.  This involves consistent models of the world.


.NEWSSS STAGES IN INCORPORATING VISUAL FEEDBACK

	There are roughly three different stages in the process
of incorporating visual feedback into AL: (1) completely separate
modules, (2) picture taking within AL but models and processing separate,
and (3) everything in AL.  These stages are briefly discussed below.

.GROUP
    COMPLETELY SEPARATE MODULES

	This means that the object modules, interpreters, camera
    control routines, etc. are in one or more modules and the AL system
    is another module.  Communication between modules is by
    messages.  This type of communication restricts the mode
of operation; feedback will only be available while the arm is not
in motion.  Motions of accommodation would not be possible.
.APART

	The current Stanford hand-eye system is of this form.  It
    will be straightforward to provide this type of system with AL.
    However, it has obvious limitations and hopefully would only
    be a temporary solution.

    PICTURE TAKING WITHIN AL

	This is the first step toward a complete integration.
AL would provide camera control, pictures, picture taking, and
    ways to call procedures which share data.  The object models could
    either be written in SAIL (and be on the PDP10) of be written in
    a PDP11 language (and be on the PDP11).  In either case the models
    and pictures would be available to external routines which analyze
    pictures and return improved location values for objects.  Visual
    servoing and dynamic feedback still could not be done;  there is no
    way to control the scheduling to insure the necessary service.

	This type of procedure-calling is designed into the current
    AL system.  It mainly involves a smart loader.  The other extensions
    are reasonably straightforward; it appears to be an easy step
    up to this type of system.  Its advantage over the previous system
    is that the basic requirements for doing visual feedback are all
    directly accessible from within one system (assuming the routines
    are on the PDP11).  This provides a chance to try out some of the
    ideas before moving on to the next stage.

    COMPLETE INTEGRATION

	Complete integration would involve motions of accommodation
in full generality, with modifications to trajectories while they
are being executed.  Picture taking and processing would all be
run under AL, and they would be interfaced into the timing scheme
to insure proper service.  Not only would true visual servoing
be possible, but also fine control of the hand based on delicate
touch sensing.

.newss DYNAMIC FRAMES 

One very desirable feature would be an ability to describe
and use continually varying variables.
In industrial applications, for instance,
the runtime system should automatically make the corrections required
to track an object on a moving conveyor.
Initially, this facility is not being implemented, although
we are studying the problems involved.  Actually, only a very
few new constructs would need to be introduced into the language
to allow such things to be described.  The principal addition
required is a way of warning the compiler that some variables
may be %4dynamic%*.  For instance,
.unfill
	DYNAMIC DISTANCE VECTOR v;
	DYNAMIC FRAME chain_hoist;
.refill
would tell the compiler that v and chain_hoist may vary continuously
with time.  Trajectories to any locations that depend on such
variables must be handled a bit differently by the servo.  Instead
of applying the brakes 
at the end of a MOVE, the servo should 
periodically recompute the destination location, obtain a new
arm solution, and then servo to the new joint values.

The normal AL graph structures are readily adapted to such dynamic
values.  Essentially all that is required is the addition of a
special reserved scalar variable TIME, which is changed every
clock tick, thus invalidating any values calculated by expressions
that depend on TIME (see {ssref gph} and {apref rgp}).  For
instance we might have
.unfill
	DYNAMIC FRAME conveyor_belt;
	VELOCITY SCALAR speed; α{%4speed of the conveyor belt%*α}
	speed α←5*IN/SEC;

	conveyor_belt <= FRAME(NILROT,speed*TIME*Y);
.maybreak

.COMT 10
      α{In this example, we won't ever use the "true" location
	of the belt.  Rather, we will affix things to it,
	so that they are carried along by the belt.α}
.END

.maybreak
	TIME SCALAR t0;
	REQUIRE "PUMP.075" SOURCE_FILE;
.COMT 10
      α{This defines, among other things, the frames
	pumpbase and pumpgrasp.  Initially, suppose that
	we know that the pumpbase is somewhere on the 
	conveyor.  We call a vision routine to find it.α}
.END

	VISUALLY_LOCATE(pumpbase,t0);
	
.COMT 10
      α{Also, set t0 to the value of TIME at which the picture 
	was taken (ie the time that the pumpbase was at the 
	location set by the procedure).α}
.END

.maybreak
	AFFIX pumpbase TO conveyor_belt
		AT (pumpbase α→ FRAME(NILROT,speed*t0*Y));

      α{%4One effect of this is:
	   %*pumpbase(t) <= (pumpbase(t0)α→conveyor_belt(t0))*conveyor_belt;
	α}
.maybreak

	MOVE YELLOW TO pumpgrasp;

.COMT 10
      α{Presumably, pumpgrasp is attached rigidly to pumpbase.
	Since pumpbase is attached to a dynamic thing (conveyor_belt)
	then pumpgrasp is computed dynamically, too, so that
	the arm will track the grasp point.α}
.END
.maybreak

	CENTER YELLOW;
         α{%4grasps the object%*α}
.maybreak
	UNFIX pumpbase FROM conveyor_belt;
	AFFIX pumpbase TO YELLOW;
	MOVE pumpbase TO jig_location_1; α{%4wherever %*that%4 is%*α}
.refill

It is perhaps worth pointing out that there is nothing particularly
magical about TIME; a similar technique could be used, say, for
moving to some frame whose value is a function of a continuously
varying A-to-D reading.

.NEWSS EXTENSIONS TO OTHER ARMS AND DEVICES

	The initial version of the AL system will be designed to run
with two  Stanford Arms,  but the system is  in no way limited to any
particular manipulators.    All manipulator-dependent  routines  are
grouped together  and are  written in  SAIL;   in order  to interface
another  manipulator to  the system these  routines would  have to be
rewritten, most notably  the solution and  dynamics models.  In  the
case of  non-manipulator type devices,  such  as cranes,
the trajectory generating routines would also need rewriting, but as
we lack any experience in this direction we will pursue it no further
at this time.

	Simple devices such as vices or tools have their own keyword
syntax and are controlled by the OPERATE statement. In this  case new
routines would need to be added.

.NEWSS FINE CONTROL

	Interactive control of the  arm has to date been  limited; we
can  output joint  torque and  monitor joint  position, and  have two
binary touch sensors inside  the fingers. Force-sensing elements  are
being developed for  the hand and we are interested  in more powerful
touch  sensors; when we have gained  experience with these devices we
will extend the language to facilitate their use. The present version
of  the language  reflects  those things  which we  have  verified in
practice and feel will move development in the right direction.

.NEWSS COLLISION AVOIDING

	Since the  available collision avoiders  are quite  slow, the
initial  system relies  upon the  user to  provide his  own collision
avoiding in  the form  of VIAs  and DEPROACHes.  When fast  collision
avoiders become  available they can  be meaningfully included  in the
system.